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FIELD OF THE INVENTION 



6 



The present invention generally relates to runtime 



7 reconf igurable systems, and more particularly to runtime 

8 reallocation of PLD resources in response to changing 

9 activity levels. 

10 

11 BACKGROUND 

12 The field of reconf igurable computing has advanced 

13 steadily for the past decade, using programmable logic 

14 devices (PLDs) as the basis for high-performance 

15 reconf igurable systems. Run-Time Reconf igurable (RTR) 

16 systems distinguish themselves by performing circuit logic 

17 customization at run-time. RTR systems using PLDs are 

18 expected to result in systems that require less hardware, 

19 less software, and fewer input/output resources than 

20 traditional ASIC or PLD-based systems. 

21 In multiprocessor data processing systems, various 

22 systems are known for balancing the work load between the 

23 processors. An advantage to such systems is that the 

24 functions implemented by the system can be software 

25 controlled, thereby providing great flexibility. However, 

26 such systems tend to be large, expensive, and complex. 

27 Furthermore, software-implemented functions tend to be 

28 slower than hardware implementations. 

29 A system and method that address the aforementioned 

30 problems, as well as other related problems, are therefore 

31 desirable. 
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1 SUMMARY OF THE INVENTION 

2 A system and method for allocating resources of 

3 programmable logic devices (PLDs) according to the activity 

4 levels of the functions implemented on the PLDs are 

5 provided. A load monitor surveys the activity levels of the 

6 functions implemented on the PLDs. When decreasing and/or 

7 increasing activity levels are detected, the PLD resources 

8 are reallocated between the various functions in proportion 

9 to the decreasing and/or increasing activity levels. 

10 In another embodiment, a reserve of PLD resources is 

11 maintained. When the activity level of one or more 

12 functions is increasing and none of the other functions have 

13 decreasing activity levels, some portion of the reserve is 

14 allocated to the functions having increasing activity 

15 levels. 

16 Various other embodiments are set forth in the Detailed 

17 Description and Claims which follow. 

18 

19 BRIEF DESCRIPTION OF THE DRAWINGS 

20 FIG. 1 is a functional block diagram of a system 100 

21 having control logic for reconfiguration of PLD resources 

22 and load monitoring of different functions that are 

23 implemented on one or more PLDs, in accordance with one 

24 embodiment of the invention. 

25 FIG. 2 is a flowchart of a process for monitoring the 

26 activity level of the functions implemented on the PLD(s). 

27 FIG. 3 A is a flowchart of a process for determining 

28 the levels at which PLD resources should be reallocated 

29 between a number of functions when it is detected that the 

30 activity level for one of the functions is decreasing. 

31 FIG. 3B is a flowchart of a process for determining 

32 the levels at which PLD resources should be allocated from 

33 the reserve of resources when the activity level of a 

34 function is increasing. 
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1 FIG. 4 is a flowchart of a process for controlling 

2 reconfiguration of PLDs in response to control parameters 

3 from the load monitor. 
4 

5 DETAILED DESCRIPTION 

6 In various embodiments, the invention monitors the 

7 activity level of various functions that are implemented on 

8 one or more PLDs . When the activity level of a function 

9 has decreased, a determination is made as to whether PLD 

10 resources of the function should be reprogrammed for 

11 another function. If so, the PLD resources are 

12 reprogrammed accordingly. 

13 FIG. 1 is a functional block diagram of a system 100 

14 having control logic for reconfiguration of PLD resources 

15 and load monitoring of different functions that are 

16 implemented on one or more PLDs, in accordance with one 

17 embodiment of the invention. The system includes one or 

18 more PLDs 102 that implement different functions, a storage 

19 element 104 for storing one or more configuration 

20 bit streams 106 that program the PLD(s) to implement the 

21 functions, a configuration control element 108, and a load 

22 monitor 110. 

23 The invention is applicable to systems having any of a 

24 variety of PLDs, for example FPGAs or CPLDs, where the 

25 particular function and the resources available on the PLD 

26 will determine the number of PLDs required. Partial 

27 reconf igurability of the PLD will also influence the number 

28 PLDs in the system. For example, the Virtex FPGA from 

29 Xilinx supports partial reconfiguration. Therefore, 

30 multiple functions may be implemented on a Virtex FPGA, 

31 depending on the resource requirements of the different 

32 functions. If it is determined that FPGA resources should 
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1 be taken from one function and devoted to another function, 

2 the FPGA can be partially reconfigured for the reallocation 

3 of resources . 



5 associated configuration bitstreams. For example, 

6 configuration bitstream 106 implements a first function, 

7 configuration bitstream 112 implements a second function, 

8 configuration bitstream 114 implements a third function, 

9 and configuration bitstream 116 implements an nth function. 

10 In one embodiment, storage 104 for the configuration 

11 bitstreams may include one or more in-system memory 

12 elements. In another embodiment, the configuration 

13 bitstreams may be stored outside the system on network 

14 storage, for example. Those skilled in the art will 

15 recognize other suitable storage alternatives for different 

16 application requirements . 

17 Load monitor 110 is coupled to the PLDs for monitoring 

18 the activity level of the functions. For example, the PLDs 

19 may implement a plurality of communications channels, 

20 wherein different ones of the channels are dedicated to 

21 different protocols. If the traffic on the channels 

22 implementing a first protocol has decreased and the traffic 

23 on the channels implementing a second protocol has 

24 increased, some of the channels for the first protocol may 

25 be reallocated for the second protocol. The load monitor 

26 not only detects the need for reprogramming but also 

27 determines which PLD resources should be reprogrammed. 

28 Load monitor 110 also signals the functions implemented on 

29 the PLDs when a reconfiguration is to occur so that work- 

30 in-process can be finished or reassigned. 

31 The mechanism by which load monitor 110 monitors the 

32 activity levels of the different functions can be adapted 
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1 in accordance with the requirements of the application. 

2 For example, the functions may be designed with a feature 

3 for reporting the activity levels via an output port of the 

4 PLDs. Alternatively, registers internal to the PLDs and 

5 readable by the load monitor may be used for storing the 

6 current activity levels. 

7 In order to reconfigure PLD resources, load monitor 

8 110 must track which functions are implemented on which PLD 

9 resources. Furthermore, if the different functions require 
10 different types and numbers of PLD resources, those too 

n must be tracked by the load monitor. 

12 Configuration control element 108 is coupled to load 

13 monitor 110, storage 104, and to the PLDs 102. When 

14 reprogramming is requested by the load monitor, 



[J 15 configuration control 108 signals the selected PLD(s) and 

?f? 

- 

y 16 initiates transfer of the selected configuration 

17 bitstream(s) from storage 104 to the appropriate PLD(s) . 

b 18 It will be appreciated that the logic of load monitor 110 

*j 19 and configuration control element 108 could be integrated 

20 into a single device in another embodiment. The particular 

S = 

fj 21 distribution of functionality between load monitor 110 and 

w 22 configuration control element 108, as well as the 

23 particular communication links to PLDs 102, can be tailored 

24 for specific application requirements. 
25 

26 FIG. 2 is a flowchart of a process for monitoring the 

27 activity level of the functions implemented on the PLD(s) . 

28 The process generally entails performing an initial 

29 allocation of PLD resources between the functions and an 

30 initial configuration of the PLDs and then monitoring 

31 activity levels of the functions and reconfiguring the PLDs 

32 as necessary. 



5 



X-651 




PATENT 



1 At step 152, an initial allocation of PLD resources is 

2 selected for the various functions. For example, in an 

3 application such as a communications switch, certain 

4 numbers of channels may be dedicated to different 

5 communications protocols. Thirty- two channels may be 

6 dedicated to protocol A, 64 channels may be dedicated to 

7 each of protocols B and C, and so on. Thus, the load 

8 monitor determines the number of channels to devote to each 

9 of the protocols. 

10 Each channel maps to a set of PLD resources . In one 

n embodiment, not all resources are initially allocated. For 

12 example, 10% of the total channels are not allocated to any 

13 protocol and are held as a reserve of resources for peak 

14 activity levels. At step 154, the configuration control is 

15 signaled to begin the initial configuration. The load 

16 monitor provides the configuration control with information 

17 identifying which PLD resources are to be configured with 

18 which configuration bitstreams. The manner in which PLD 

19 resources and configuration bitstreams are identified will 

20 depend on the specific functions and the capabilities of 

21 the particular PLD. Furthermore, if a single PLD supports 

22 more than one function, then a number of configuration 

23 bitstreams may be required to cover the possible 

24 combinations for allocating resources of the PLD between 

25 the different functions. 

26 Once the PLDs have been configured and the system is 

27 operational, at step 156, the load monitor establishes 

28 communications with the functions on the PLDs. At step 

29 158, the load monitor begins monitoring the activity levels 

30 of the functions. The load monitor samples (step 160) the 

31 activity levels every t interval of time for each function. 

32 A typical measure of activity level for a communications 
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protocol channel is the number of packets processed per 
second. If a protocol is supported across multiple 
channels, the activity level is the total activity across 
all the channels implementing the protocol. 

Decision step 162 tests whether the activity level of 
any of the functions is decreasing. When a function shows 
a decreasing activity level over n samples and the overall 
decrease is greater than a given threshold, some of the 
resources allocated to the function are freed for re- 
allocation to other applications. The value of n and the 
threshold level are chosen so that reallocation is not 
performed in response to random and instantaneous changes. 
Each function may have a different threshold that varies 
based on the number of resources allocated to the function 
since functions having greater numbers of resources will be 
able to accommodate greater activity levels. In various 
embodiments, the functions implemented on the PLDs are 
programmed to periodically report activity levels to the 
load monitor either via an on-board PLD register or by 
transmission to the load monitor. 

Step 164 determines how resources are to be 
reallocated between the functions implemented on the PLDs 
(FIG. 3A describes how the reallocated resource levels are 
determined for decreasing activity levels) . At step 166, 
the load monitor signals the PLD(s) that will be 
reconfigured to complete work-in-process prior to 
reconfiguration. The work may be completed by virtue of 
finishing the work or transferring the work to a different 
resource (e.g., another PLD that will not be reconfigured). 
For I/O tasks on resources to be freed, a number of steps 
are performed: 
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1 . The output ports from the resources to be freed are 
connected to output ports of another resource that 
implements the same function. For example, if 
channels A and B both implement protocol 1 and the 
resources from channel A are to be freed, the output 
ports of channel A are connected to unassigned 
output ports of channel B, or if time-slicing is 
used, to outputs of channel B having unused 
capacity. 

2 . Direct input data from the resource to be freed to 
another resource that implements the same function. 
For example, (continuing the example of step 1) the 
input data directed at channel A is redirected to 
the input ports of channel B. 

3 . Disable the output ports of the resource to be 
freed. 

4. Disable the input ports of the resource to be freed. 

The connection and redirection of inputs and outputs can 
best be facilitated by either special I/O circuitry (for 
example, a crossbar switch) , or a PLD reconfigured to re- 
map input and output connections. 

At step 168, the configuration control element is 
signaled to begin the reconfiguration. The PLD resources 
that are reconfigured are those identified at step 164. 

If decision step 162 finds that none of the functions 
have decreasing activity levels, the process is directed to 
decision step 172 to test whether any of the functions have 
increasing activity levels. If there are no functions 
having decreasing or increasing activity levels, the 
process returns to sampling the activity levels at step 
160. 
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If any of the functions are detected as having 
increased activity levels, the process is directed to step 
174. When a function shows an increasing activity level 
over n samples and the overall increase is greater than a 
given threshold, resources from the reserve a allocated to 
the function (s) . The values of n and the threshold level 
are chosen so that reallocation is not performed in 
response to random and instantaneous changes. The 
thresholds associated with increasing activity levels may 
be different from the thresholds associated with decreasing 
activity levels, and each function may have a different 
threshold that varies based on the number of resources 
allocated to the function since functions having greater 
numbers of resources will be able to accommodate greater 
activity levels. 

At step 174, it is determined how resources from the 
reserve are to be allocated to the function (s) (FIG. 3B 
describes how the reallocated resource levels are 
determined for increasing activity levels) . The process 
then continues at step 168 as described above. 

FIG. 3A is a flowchart of a process for determining 
the levels at which PLD resources should be reallocated 
between a number of functions when the activity level of a 
function is decreasing. The number of resources to free 
from a function is determined at step 202 and is based on 
the following model. Each resource can service a maximum 
load of R L for its given function. Thus, if M resources are 
assigned to a function, then the maximum sustainable 
activity level is M*R L . If the load monitor determines that 
the activity level has decreased such that a current 
activity level = (M-P) * R L (P<=M) , then the (P-l) resources 
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are freed (one is left in reserve with the function) . At 
step 204, the number of resources of the function is 
reduced by the amount determined at step 202. For example, 
the number of channels that implement a particular protocol 
is reduced. 

Decision step 206 determines whether the freed 
resources should be allocated to other functions or held in 
reserve. The decision is made based on a projection of 
future activity levels of the other functions. The 
projection can be linear, least squares, or another 
technique that best models the overall system behavior. 
In one embodiment, the -projection is made based on past 
behavior. Sample activity levels are taken by the load 
monitor. The values associated with the activity levels 
are stored over a sufficiently long window of time. The 
length of the window is based on the usage environment. 
Past performance is modeled statistically to predict future 
behavior. If the predicted future behavior is either a 
decrease or increase in the activity level, then 
reconfiguration and re-assignment may be initiated. 

At step 208, the allocation of freed resources to 
functions is determined based on the projected, 
proportional needs of the functions. For example, if the 
activity level of a first function is projected to increase 
by 30% and the activity level of a second function is 
projected to increase by 10%, then the ratio of the 
projected increases is used when making the allocation. 
Assuming both the first and second functions begin with the 
same initial resource allocation, then the first function 
will be allocated three-fourths of the resources and the 
second function is allocated one-fourth of the resources. 
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1 If none of the activity levels is increasing, decision 

2 step 206 directs control to step 210, and the freed 

3 resources are added to the reserve of free resources. 

4 Parameters that describe the resources to allocate to the 

5 different functions are then returned to the process of 

6 FIG. 2. 
7 



8 FIG. 3B is a flowchart of a process for determining 

9 the levels at which PLD resources should be allocated from 
10 the reserve when the activity level of a function is 

n increasing. At step 222, the number of resources to remove 

12 from the reserve and allocate to the one or more functions 

£3 13 is determined. 

ij 

Irk 14 The allocation of resources takes into account the 

r ? 15 increasing needs of the entire system (i.e., across 

Lfl 

|y 16 multiple functions) . The allocation process also ensures 

H 17 that the entire reserve is not depleted so that future 

s 18 increased activity levels can be handled. In one 

C3 

vj 19 embodiment, the following model is used to manage the 

r 1 1 

'Z 20 reserve of resources: 

C3 21 1. The load monitor tracks the total percentage of 

" 22 resources allocated to each function and the 

23 percentage of resources presently in the reserve. The 

24 total of all allocated resources is 100% 

25 (Ri+R2+R3+R4+.«+Rreserve = 100%, where R x is the percent 

26 resources allocated to function X) . 

27 2. The load monitor tracks the predicted increase (or 

28 decrease) of resources for each function. Each 

29 predicted increase (or decrease) is represented as a 

30 percentage of the current resources allocated to a 

31 function. 
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When allocating from the reserve, a percentage of the 
reserve is maintained for future allocations. The portion 
of the reserve maintained for future allocations can be 
established as a fixed percentage or a sliding percentage 
that varies as a function of the activity levels of all the 
functions, for example, a smaller percentage when the 
functions are more active and a larger percentage when the 
functions are less active. The resources in the reserve 
will slowly decline to zero as the system reaches its 
saturation capacity. When the reserve is empty, only 
decreases in the activity levels of functions can trigger 
reallocations . 

Resources are proportionally allocated from the 
reserve based on the projected increased activity levels of 
the functions. In addition, functions having the greatest 
projected increase are considered first. For example, if 
function A's activity level is projected to increase 30%, 
function B's 20%, function C's 10%, and 50 units of the 
reserve are eligible to allocate, then function A should 
maximally get 25 units (1/2 * 50 units), function B should 
maximally get 17 units (1/3 * 50), and function C should 
maximally get 8 units (1/6 * 50) . 

After the maximum allocations have been established, 
the allocations and assignment priority are evaluated 
against the actual predicted unit increase. For example, 
if function A is projected to have a 30% increase but 
currently has 10 resource units, then an allocation of 3 
units rather than 25 would suffice. Thus, function A is 
allocated the 3 units and the difference is left in the 
reserve and eligible for allocation. This allows 
allocation to function B to be as much as needed. 
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1 allocated all the resources necessary to meet the 

2 projections, then the performance level of the functions 

3 may be impaired. 

4 After the selected resources of the PLD have been 

5 reconfigured in accordance with the selected allocation of 

6 resources, the load monitor continues to sample activity. 

7 At step 224, the number of resources maintained in the 

8 reserve is reduced by the amount determined at step 222. 

9 For example, the number of channels that implement a 

10 particular protocol is reduced. 

11 At step 228, the allocation of resources removed from 

12 the reserve amongst one or more functions is determined 

C3 13 based on the projected, proportional activity levels of the 

}i 14 functions. The projected activity levels and proportional 

[y 15 allocation are determined as described above in association 

tj 16 with FIG. 3A. Parameters that describe the resources to 

hi 17 allocate to the different functions are then returned to 

f : 

TSSST 

s 18 the process of FIG. 2. 

fiJ 20 FIG. 4 is a flowchart of a process for controlling 

□ 21 reconfiguration of PLDs in response to control parameters 

™ 22 from the load monitor. 

23 At step 252, one or more configuration bitstreams are 

24 selected in response to the signals from the load monitor. 

25 The particular configuration bitstreams that are selected 

26 will depend on the PLD resources to be reconfigured and the 

27 selected functions. 

28 At step 254, the PLD resources that are to be 

29 reconfigured are selected. 

30 The selected configuration bitstreams are then 

31 downloaded to the PLDs at step 256. 
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1 At step 258, the reconfigured PLDs are restarted with 

2 the reprogrammed functions . 

3 Finally, at step 260 the load monitor is signaled that 

4 the reconfiguration is complete. 

5 The present invention is believed to be applicable to 

6 a variety of systems having a plurality of PLD- implemented 

7 functions. Other aspects and embodiments of the present 

8 invention will be apparent to those skilled in the art from 

9 consideration of the specification and practice of the 
10 invention disclosed herein. It is intended that the 

n specification and illustrated embodiments be considered as 

12 examples only, with a true scope and spirit of the 

C3 13 invention being indicated by the following claims. 
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